Drop-off location planning for delivery vehicle

ABSTRACT

An example method may include receiving, from a client computing device, an indication of a target drop-off spot for an object within a first virtual model of a first region of a delivery destination. A second virtual model of a second region of the delivery destination may be determined based on sensor data received from one or more sensors on a delivery vehicle. A mapping may be determined between physical features represented in the first virtual model and physical features represented in the second virtual model to determine an overlapping region between the first and second virtual models. A position of the target drop-off spot within the second virtual model may be determined based on the overlapping region. Based on the position of the target drop-off spot within the second virtual model, the delivery vehicle may be navigated to the target drop-off spot to drop off the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of, and claims priorityto, U.S. patent application Ser. No. 16/429,842, titled “Drop-offLocation Planning for Delivery Vehicle,” filed on Jun. 3, 2019, which isa continuation application of, and claims priority to, U.S. patentapplication Ser. No. 15/295,995, now U.S. Pat. No. 10,353,388, titled“Drop-off Location Planning for Delivery Vehicle,” filed on Oct. 17,2016. The disclosures of the foregoing applications are incorporatedherein by reference in their entirety for all purposes.

BACKGROUND

An unmanned vehicle, which may also be referred to as an autonomousvehicle, is a vehicle capable of travel without a physically-presenthuman operator. An unmanned vehicle may operate in a remote-controlmode, in an autonomous mode, or in a partially autonomous mode.

When an unmanned vehicle operates in a remote-control mode, a pilot ordriver that is at a remote location can control the unmanned vehicle viacommands that are sent to the unmanned vehicle via a wireless link. Whenthe unmanned vehicle operates in autonomous mode, the unmanned vehicletypically moves based on pre-programmed navigation waypoints, dynamicautomation systems, or a combination thereof. Further, some unmannedvehicles can operate in both a remote-control mode and an autonomousmode, and in some instances may do so concurrently. For instance, aremote pilot or driver may wish to leave navigation to an autonomoussystem while manually performing another task, such as operating amechanical system for picking up or dropping off objects, as an example.

Various types of unmanned vehicles exist for different environments. Forinstance, unmanned vehicles exist for operation in the air, on theground, underwater, and in space. Examples include quad-copters andtail-sitter UAVs, among others. Unmanned vehicles also exist for hybridoperations in which multi-environment operation is possible. Examples ofhybrid unmanned vehicles include an amphibious craft that is capable ofoperation on land as well as on water or a floatplane that is capable oflanding on water as well as on land. Other examples are possible.

SUMMARY

Example embodiments include systems and operations for delivery ofobjects (e.g., packages) to a target drop-off spot within a deliverydestination. A client computing device (e.g., phone, tablet, or othercomputing device equipped with or connectable to sensors) may be used tocollect sensor data representing a first region of the deliverydestination. A first virtual model may be constructed from the sensordata and the client computing device may be used to designate the targetdrop-off spot within the delivery destination by placing a target markerin the first virtual model. In response to a request for packagedelivery, an unmanned delivery vehicle may be dispatched with thepackage to the delivery destination. Upon arriving at the generalvicinity of the delivery destination, the delivery vehicle may begincollecting sensor data to locate the target drop-off spot within thedelivery destination. In particular, a second virtual model of a secondregion of the delivery destination may be generated from the sensor dataof the delivery vehicle. A mapping may be determined between physicalfeatures represented in the first and second virtual models to determinean overlapping region between the first and second virtual models. Theoverlapping region may be used to align the first model with the secondmodel and thus determine a position of the target drop-off spot withinthe second virtual model, as perceived by the delivery vehicle.Accordingly, the delivery vehicle may then be navigated to the targetdrop-off spot to place the package at the target drop-off spot.

In one example, a method is provided that includes receiving, from aclient computing device, an indication of a target drop-off spot for anobject within a first virtual model of a first region of a deliverydestination. The first virtual model represents first physical featuresof the first region of the delivery destination. The method alsoincludes receiving, from one or more sensors on a delivery vehicle,sensor data indicative of a second region of the delivery destination.The method additionally includes determining, based on the sensor data,a second virtual model of the second region of the delivery destination.The second virtual model represents second physical features of thesecond region on the delivery destination. The method further includesdetermining a mapping between one or more of the first physical featuresand one or more of the second physical features to determine anoverlapping region between the first virtual model and the secondvirtual model. The method yet further includes determining, based on theoverlapping region, a position of the target drop-off spot within thesecond virtual model. Finally, the method includes providinginstructions to navigate, based on the position of the target drop-offspot within the second virtual model, the delivery vehicle to the targetdrop-off spot to place the object at the target drop-off spot.

In another example a system is provided that includes a deliveryvehicle, one or more sensors connected to the delivery vehicle, and acontrol system. The control system is configured to receive anindication of a target drop-off spot for an object within a firstvirtual model of a first region of a delivery destination. The firstvirtual model represents first physical features of the first region ofthe delivery destination. The control system is also configured toreceive, from the one or more sensors, sensor data indicative of asecond region of the delivery destination. The control system isadditionally configured to determine, based on the sensor data, a secondvirtual model of the second region of the delivery destination. Thesecond virtual model represents second physical features of the secondregion on the delivery destination. The control system is furtherconfigured to determine a mapping between one or more of the firstphysical features and one or more of the second physical features todetermine an overlapping region between the first virtual model and thesecond virtual model. The control system is yet further configured todetermine a position of the target drop-off spot within the secondvirtual model based on the overlapping region. Finally, the controlsystem is configured to provide instructions to navigate, based on theposition of the target drop-off spot within the second virtual model,the delivery vehicle to the target drop-off spot to place the object atthe target drop-off spot.

In an additional example, a non-transitory computer-readable storagemedium is provided having stored thereon instructions that, whenexecuted by a computing device, cause the computing device to performoperations. The operations include receiving, from a client computingdevice, an indication of a target drop-off spot for an object within afirst virtual model of a first region of a delivery destination. Thefirst virtual model represents first physical features of the firstregion of the delivery destination. The operations also includereceiving, from one or more sensors on a delivery vehicle, sensor dataindicative of a second region of the delivery destination. Theoperations additionally include determining, based on the sensor data, asecond virtual model of the second region of the delivery destination.The second virtual model represents second physical features of thesecond region on the delivery destination. The operations furtherinclude determining a mapping between one or more of the first physicalfeatures and one or more of the second physical features to determine anoverlapping region between the first virtual model and the secondvirtual model. The operations yet further include determining, based onthe overlapping region, a position of the target drop-off spot withinthe second virtual model. Finally, the operations include providinginstructions to navigate, based on the position of the target drop-offspot within the second virtual model, the delivery vehicle to the targetdrop-off spot to place the object at the target drop-off spot.

In a further example, a system is provided that includes means forreceiving, from a client computing device, an indication of a targetdrop-off spot for an object within a first virtual model of a firstregion of a delivery destination. The first virtual model representsfirst physical features of the first region of the delivery destination.The system also includes means for receiving, from one or more sensorson a delivery vehicle, sensor data indicative of a second region of thedelivery destination. The system additionally includes means fordetermining, based on the sensor data, a second virtual model of thesecond region of the delivery destination. The second virtual modelrepresents second physical features of the second region on the deliverydestination. The system further includes means for determining a mappingbetween one or more of the first physical features and one or more ofthe second physical features to determine an overlapping region betweenthe first virtual model and the second virtual model. The system yetfurther includes means for determining, based on the overlapping region,a position of the target drop-off spot within the second virtual model.Finally, the system includes means for providing instructions tonavigate, based on the position of the target drop-off spot within thesecond virtual model, the delivery vehicle to the target drop-off spotto place the object at the target drop-off spot.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an unmanned aerial vehicle, according to an exampleembodiment.

FIG. 1B illustrates an unmanned aerial vehicle, according to an exampleembodiment.

FIG. 1C illustrates an unmanned aerial vehicle, according to an exampleembodiment.

FIG. 1D illustrates an unmanned aerial vehicle, according to an exampleembodiment.

FIG. 1E illustrates an unmanned aerial vehicle, according to an exampleembodiment.

FIG. 2 illustrates a block diagram of components of an unmanned deliveryvehicle, according to an example embodiment.

FIG. 3 is a simplified block diagram illustrating an unmanned deliveryvehicle system, according to an example embodiment.

FIG. 4 illustrates an example flow diagram, according to an exampleembodiment.

FIGS. 5A, 5B, and 5C illustrate a client computing device capturingsensor data representing a delivery destination, according to an exampleembodiment.

FIG. 6 illustrates an example designation of a target drop-off spotwithin an example virtual model of a delivery destination, according toan example embodiment.

FIG. 7A illustrates a delivery vehicle scanning a delivery destination,according to an example embodiment.

FIG. 7B illustrates another example of a virtual model of a deliverydestination, according to an example embodiment.

FIG. 8A illustrates an example mapping between two virtual models,according to an example embodiment.

FIG. 8B illustrates delivery of an object by a delivery vehicle to atarget drop-off spot within a delivery destination, according to anexample embodiment.

FIG. 9 illustrates an example virtual model, according to an exampleembodiment.

DETAILED DESCRIPTION

The following detailed description describes various features andoperations of the disclosed devices, systems, and methods with referenceto the accompanying figures. The illustrative device, system, and methodembodiments described herein are not meant to be limiting. It should beunderstood that the words “exemplary,” “example,” and “illustrative,”are used herein to mean “serving as an example, instance, orillustration.” Any implementation, embodiment, or feature describedherein as “exemplary,” “example,” or “illustrative,” is not necessarilyto be construed as preferred or advantageous over other implementations,embodiments, or features. Further, aspects of the present disclosure, asgenerally described herein and illustrated in the figures, can bearranged, substituted, combined, separated, and designed in a widevariety of different configurations.

In the following detailed description, reference is made to theaccompanying figures, which form a part thereof. In the figures, similarsymbols typically identify similar components, unless context dictatesotherwise. Other embodiments may be utilized, and other changes may bemade, without departing from the spirit or scope of the subject matterpresented herein. Further, unless otherwise noted, figures are not drawnto scale and are used for illustrative purposes only. Moreover, thefigures are representational only and not all components are shown. Forexample, additional structural or restraining components might not beshown.

I. OVERVIEW

Package delivery services currently rely on human delivery people to usetheir judgment and intuition in finding a delivery destination (e.g.,home or work address) and selecting a spot at the delivery destinationin which to place a package upon delivery (e.g., front porch, side door,etc.). As package delivery services begin to employ unmanned deliveryvehicles, package recipients may have more control over when packagesare delivered, the delivery destination to which packages are delivered,as well as the spot at which packages are placed upon delivery. Forexample, in addition to the existing paradigm of delivery to home orwork, unmanned delivery vehicles may be scheduled to deliver packages tolocations other than a recipient's home (e.g., current user locationsuch as a cafe, restaurant, gym, etc.) at precise, user-specified times.Additionally, recipients may be able to designate a precise targetdrop-off spot for a package. For instance, recipients may specify spotswhere the package will not be at risk of theft and will not block doorsor walkways.

In an example embodiment, a client computing device (e.g., phone,tablet, or other computing device equipped with or connectable tosensors) may be used to designate a target drop-off spot for a packagewithin a delivery destination (i.e., the target spot may be a smallersubset of the delivery destination). Specifically, the package recipientmay use the client computing device to capture video or photos of aregion of the delivery destination to which the package is to bedelivered. For example, the recipient may capture a video or a series ofphotos of a front porch of the recipient's home. Each photo or videoframe may be associated with additional sensor data from sensors on orconnected to the client computing device such as gyroscopes,magnetometers, accelerometers, and global positioning system (GPS)module.

The captured data may be used to generate a virtual model representingthe region of the delivery destination to which the package is to bedelivered. The virtual model may contain representations of physicalfeatures of the delivery destination (e.g., trees, bushes, windows,doors, stairs, neighboring homes, etc.). The virtual model may bedisplayed by way of a user interface of the client computing device andthe user may be prompted to designate a target drop-off spot within thevirtual model.

The client computing device may receive input indicating selection ordesignation of a spot within the virtual model. For example, thedesignation may be visualized by displaying a “bullseye” target markerat the designated spot within the virtual model. The location of thetarget marker within the virtual model may indicate a physical locationat which the user prefers to have the package dropped off upon delivery.The marker may be movable within the virtual model and may “snap” tosurfaces represented within the model. In some embodiments, the targetdrop-off spot may be designated directly within one or more of thecaptured images or video frames, prior to generation of the model. Theoperations of model generation may, based on the target drop-off spotdesignation within each image, determine the position of the targetdrop-off spot within the virtual model.

Upon marker placement, a validation may be performed to determinewhether the delivery vehicle is capable of navigating to the targetdrop-off spot and whether the package, based on anticipated size andshape of the package, is expected to fit and/or rest securely in thetarget drop-off spot.

In some instances, a virtual model may be available from a prior packagedelivery to a selected delivery destination. Accordingly, the operationsfor generating the virtual model may be omitted and the user may proceedwith designating a target drop-off spot within an existing model orselecting a previously designated target drop-off spot within the model.

The virtual model and the indication of the target drop-off spot withinthe model may be transmitted to a server device, to a delivery vehicle,and/or may be stored locally on the client computing device. A deliveryvehicle (e.g., aerial drone, ground vehicle, water-based vehicle, or ahybrid thereof) may be dispatched to deliver the requested package tothe delivery destination. The delivery vehicle may navigate to thegeneral vicinity of the delivery destination based on GPS information.When the delivery vehicle arrives within a threshold distance of the GPScoordinates of the delivery destination, GPS information may lack thenecessary accuracy to allow for navigation of the vehicle to the targetdrop-off spot within the delivery destination.

Accordingly, the delivery vehicle may begin capturing sensor data fromsensors on the delivery vehicle (e.g., cameras, depth sensors, lightdetection and ranging (LIDAR) device, accelerometers, magnetometers,gyroscopes, etc.) to generate a second virtual model that the vehiclewill use to locate and navigate to the target drop-off spot. Thedelivery vehicle may include some or all of the sensors that areincluded on the client computing device. Additionally, the deliveryvehicle may include additional sensors that aid in vehicle navigationand may be used to generate the second virtual model (e.g., altimetersin the case of an aerial vehicle).

A second virtual model may be generated based on the sensor data fromthe delivery vehicle. A mapping may be determined of representations ofphysical features between the first virtual model (e.g., theuser-specified model) and the second virtual model. The mapping may beused to determine an overlapping region between the first model and thesecond model. Stated another way, the mapping may be used to spatiallyalign the second model with the first model to identify a region of thedelivery destination common to both models. In some embodiments, themapping may be determined by directly comparing the first model to thesecond model. Alternatively, the mapping may be determined by comparingeach of the first model and the second model to a master virtual modelrepresenting the geographic area surrounding the delivery destination.Based on the overlapping region, the system may determine the positionof the target drop-off spot within the second virtual model.

In some instances, before a complete second model is determined, theinitial portions of the second model might not yet represent the regionof the delivery destination containing the target drop-off spot.However, the second model may overlap with a portion of the first modelthat does not represent the target drop-off spot. Based on thisoverlap/alignment, the control system of the delivery vehicle maydetermine to navigate the vehicle to scan the region of the environmentestimated to contain the target drop-off spot. In response, the secondmodel may be expanded and updated to represent the region containing thetarget drop-off spot and, based on the mapping between the models, aposition of the target drop-off spot within the second virtual model maybe determined.

In some instances, due to GPS positioning error, the delivery vehiclemay initially scan a region next to the delivery destination that doesnot overlap with the region of the delivery destination scanned by theuser of the client computing device. Accordingly, the system might notbe able to determine a mapping and may cause the delivery vehicle tocontinue systematically scanning other regions near the delivery vehicleuntil a region is found that overlaps with the user-specified region(e.g., region represented by the first virtual model).

Alternatively or additionally, a master virtual model or map may be usedto determine the mapping between the first and second virtual models.Namely, a mapping may be determined between the master model and each ofthe first and second models. The determined mapping may be used tolocate each of the first and second models within the master model andthus determine the geographic positioning of the second model withrespect to the first model. Accordingly, instead of systematicallyscanning the area to find an overlapping region, the system may use themaster model to determine a direction or trajectory along which tonavigate the delivery vehicle to locate the delivery destination and thetarget drop-off spot therein. While navigating along the determinedtrajectory, the delivery vehicle may scan portions of the deliverydestination to build up the second virtual model to represent the targetdrop-off spot within the second virtual model. Thus, based on therelative geographic positioning of the first and second models, thesystem may locate the target drop-off spot within the second virtualmodel.

The master model may be generated and updated based onpreviously-generated virtual models of various delivery destinationswithin a geographic region surrounding the current delivery destination.The master model may, in some examples, be used as a backup when amapping cannot be determined directly between the first and secondvirtual models. In particular, when the control system is unable to mapa region scanned by the delivery vehicle to the environment scanned bythe client computing device, the system may attempt to map both regionsto the master model of the geographic location surrounding the deliverydestination. A successful mapping may allow the system to determine therelative spatial positioning between the region scanned by the clientcomputing device and the region scanned by the delivery vehicle. Thus,the system may determine a direction in which to navigate the deliveryvehicle to find the delivery destination.

For example, due to GPS error, the delivery vehicle may initially scan ahouse to the east of the delivery destination. The system might beunable to map physical features of the scanned house to physicalfeatures within the region specified by the client computing device.However, a mapping may be determined between physical features of thescanned house and a portion of the master model representing the scannedhouse. Accordingly, the system may determine that the delivery vehicleis too far east of the delivery destination and may cause the deliveryvehicle to navigate west. Thus, the delivery destination may besuccessfully scanned and a mapping may be determined between the firstand second virtual models. Alternatively, in some embodiments, themaster model may be used continuously in the mapping operations.

The mapping and the overlapping region between the first and secondmodels may be used to determine a position of the target drop-off spotwithin the second virtual model, regardless of whether a master model isemployed. The position of the target drop-off spot may be used todetermine the relative spatial positioning between the delivery vehicleand the target drop-off spot. Thus, the delivery vehicle may benavigated to the target drop-off spot to place the package at the targetdrop-off spot, as requested by the package recipient. In someembodiments, a human operator may take control of the delivery vehicleremotely and may navigate the vehicle to the target drop-off spot. Thehuman operator may be presented with visual cues indicating where thesystem is expecting the delivery destination and the target drop-offspot to be located. Thus, the operations herein described may be used toaid the human operator's decision-making in navigating the deliveryvehicle. Additionally, the operations herein described may be localizedto or distributed between any combination of the client computingdevice, remote server devices, and the delivery vehicles.

Further, in some embodiments, the delivery vehicle may be replaced by asecond client computing device associated with a delivery person.Instead of navigating the delivery vehicle, the operations hereindescribed may be used to guide the delivery person to the targetdrop-off spot designated by the package recipient. Namely, the secondclient computing device may display visual cues to guide the deliveryperson in scanning the delivery destination to generate the secondmodel. Once the second model is generated, the position of the targetdrop-off spot may be determined and may be displayed by the secondclient computing device as a virtual-reality marker to indicate to thedelivery person where to place the package. For example, the secondcomputing device may display a live video feed from the camera of thedevice and, when the camera is panned over the target drop-off spot, amarker may be displayed in the video feed indicating the target drop-offspot. Accordingly, packages may be placed in safer and/or moreconvenient locations by delivery people.

II. EXAMPLE UNMANNED DELIVERY VEHICLE SYSTEMS

Herein, the terms “unmanned delivery vehicle” and “UDV” refer to anyautonomous or semi-autonomous vehicle that is capable of performing somefunctions without a physically present human pilot.

A UDV can take various forms. A UDV may be a ground vehicle, an aerialvehicle, a water vehicle, or a multi-terrain hybrid. For example, aground UDV may take the form of a ground vehicle that includes a baseadapted for ground movement by way of wheels, tracks, or legs. Theground UDV may include a robotic arm to allow the UDV to move payload toand from the UDV. In another example, an aerial UDV may take the form ofa fixed-wing aircraft, a glider aircraft, a tail-sitter aircraft, a jetaircraft, a ducted fan aircraft, a lighter-than-air dirigible such as ablimp or steerable balloon, a rotorcraft such as a helicopter ormulticopter, and/or an ornithopter, among other possibilities. Further,the terms “drone,” “unmanned aerial vehicle” (UAV), “unmanned aerialvehicle system” (UAVS), or “unmanned aerial system” (UAS) may also beused to refer to a UDV. In a further example, a water-based UDV may takethe form of a boat, hovercraft, or submarine, among other possibilities.

FIG. 1A illustrates a simplified top-down view of a UAV, according to anexample embodiment. In particular, FIG. 1A shows an example of afixed-wing UAV 100, which may also be referred to as an airplane, anaeroplane, a biplane, a glider, or a plane, among other possibilities.The fixed-wing UAV 100, as the name implies, has stationary wings 102that generate lift based on the wing shape and the vehicle's forwardairspeed. For instance, the two wings 102 may have an airfoil-shapedcross section to produce an aerodynamic force on the UAV 100.

As depicted, the fixed-wing UAV 100 may include a wing body 104 ratherthan a clearly defined fuselage. The wing body 104 may contain, forexample, control electronics such as an inertial measurement unit (IMU)and/or an electronic speed controller, batteries, other sensors, and/ora payload, among other possibilities. The illustrative UAV 100 may alsoinclude landing gear (not shown) to assist with controlled take-offs andlandings. In other embodiments, other types of UAVs without landing gearare also possible.

The UAV 100 further includes propulsion units 106, which can eachinclude a motor, shaft, and propeller, for propelling the UAV 100.Vertical stabilizers 108 (or fins) may also be attached to the wing body104 and/or the wings 102 to stabilize the UAV's yaw (i.e., left or rightturn) during flight. In some embodiments, the UAV 100 may be also beconfigured to function as a glider. To do so, UAV 100 may power off itsmotor, propulsion units, etc., and glide for a period of time.

During flight, the UAV 100 may control the direction and/or speed of itsmovement by controlling its pitch, roll, yaw, and/or altitude. Forexample, the vertical stabilizers 108 may include one or more ruddersfor controlling the UAV's yaw, and the wings 102 may include one or moreelevators for controlling the UAV's pitch and/or one or more aileronsfor controlling the UAV's roll. As another example, increasing ordecreasing the speed of all the propellers simultaneously can result inthe UAV 100 increasing or decreasing its altitude, respectively.

Similarly, FIG. 1B shows another example of a fixed-wing UAV 120. Thefixed-wing UAV 120 includes a fuselage 122, two wings 124 with anairfoil-shaped cross section to provide lift for the UAV 120, a verticalstabilizer 126 (or fin) to stabilize the plane's yaw (i.e., left orright turn), a horizontal stabilizer 128 (also referred to as anelevator or tailplane) to stabilize pitch (tilt up or down), landinggear 130, and a propulsion unit 132, which can include a motor, shaft,and propeller.

FIG. 1C shows an example of a UAV 140 with a propeller in a pusherconfiguration. The term “pusher” refers to the fact that a propulsionunit 142 is mounted at the back of the UAV and “pushes” the vehicleforward, in contrast to the propulsion unit being mounted at the frontof the UAV. Similar to the description provided for FIGS. 1A and 1B,FIG. 1C depicts common structures used in a pusher plane, including afuselage 144, two wings 146, vertical stabilizers 148, and thepropulsion unit 142, which can include a motor, shaft, and propeller.

FIG. 1D shows an example of a tail-sitter UAV 160. In the illustratedexample, the tail-sitter UAV 160 has fixed wings 162 to provide lift andallow the UAV 160 to glide horizontally (e.g., along the x-axis, in aposition that is approximately perpendicular to the position shown inFIG. 1D). Additionally, the fixed wings 162 also allow the tail-sitterUAV 160 to take off and land vertically.

For example, at a launch site, the tail-sitter UAV 160 may be positionedvertically (as shown) with its fins 164 and/or wings 162 resting on theground and stabilizing the UAV 160 in the vertical position. Thetail-sitter UAV 160 may then take off by operating its propellers 166 togenerate an upward thrust (e.g., a thrust that is generally along they-axis). Once at a suitable altitude, the tail-sitter UAV 160 may useits flaps 168 to reorient itself in a horizontal position, such that itsfuselage 170 is closer to being aligned with the x-axis than the y-axis.Positioned horizontally, the propellers 166 may provide forward thrustso that the tail-sitter UAV 160 can fly in a similar manner as a typicalairplane.

Many variations on the illustrated fixed-wing UAVs are possible. Forinstance, fixed-wing UAVs may include more or fewer propellers, and/ormay utilize a ducted fan or multiple ducted fans for propulsion.Further, UAVs with more wings (e.g., an “x-wing” configuration with fourwings), with fewer wings, or even with no wings, are also possible.

As noted above, some embodiments may involve other types of UAVs, inaddition to or in the alternative to fixed-wing UAVs. For instance, FIG.1E shows an example of a rotorcraft that is commonly referred to as amulticopter 180. The multicopter 180 may also be referred to as aquadcopter, as it includes four rotors 182. It should be understood thatexample embodiments may involve a rotorcraft with more or fewer rotorsthan the multicopter 180. For example, a helicopter typically has tworotors. Other examples with three or more rotors are possible as well.Herein, the term “multicopter” refers to any rotorcraft having more thantwo rotors, and the term “helicopter” refers to rotorcraft having tworotors.

Referring to the multicopter 180 in greater detail, the four rotors 182provide propulsion and maneuverability for the multicopter 180. Morespecifically, each rotor 182 includes blades that are attached to amotor 184. Configured as such, the rotors 182 may allow the multicopter180 to take off and land vertically, to maneuver in any direction,and/or to hover. Further, the pitch of the blades may be adjusted as agroup and/or differentially, and may allow the multicopter 180 tocontrol its pitch, roll, yaw, and/or altitude.

It should be understood that references herein to an “unmanned” deliveryvehicle or UDV can apply equally to autonomous and semi-autonomousvehicles. In an autonomous implementation, all functionality of thevehicle is automated; e.g., pre-programmed or controlled via real-timecomputer functionality that responds to input from various sensorsand/or pre-determined information. In a semi-autonomous implementation,some functions of the vehicle may be controlled by a human operator,while other functions are carried out autonomously. Further, in someembodiments, a UDV may be configured to allow a remote operator to takeover functions that can otherwise be controlled autonomously by the UDV.Yet further, a given type of function may be controlled remotely at onelevel of abstraction and performed autonomously at another level ofabstraction. For example, a remote operator could control high levelnavigation decisions for a UDV, such as by specifying that the UDVshould travel from one location to another (e.g., from a warehouse in asuburban area to a delivery address in a nearby city), while the UDV'snavigation system autonomously controls more fine-grained navigationdecisions, such as the specific route to take between the two locations,specific flight controls to achieve the route and avoid obstacles whilenavigating the route, and so on.

More generally, it should be understood that the example UDVs describedherein are not intended to be limiting. Example embodiments may relateto, be implemented within, or take the form of any type of unmanneddelivery vehicle.

III. EXAMPLE UDV COMPONENTS

FIG. 2 is a simplified block diagram illustrating components of aunmanned delivery vehicle 200, according to an example embodiment. UDV200 may take the form of, or be similar in form to, one of the UDVs 100,120, 140, 160, 180, and 190 described in reference to FIGS. 1A-1E.However, UDV 200 may also take other forms.

UDV 200 may include various types of sensors, and may include acomputing system configured to provide the functionality describedherein. In the illustrated embodiment, the sensors of UDV 200 include aninertial measurement unit (IMU) 202, ultrasonic sensor(s) 204, and a GPS206, among other possible sensors and sensing systems.

In the illustrated embodiment, UDV 200 also includes one or moreprocessors 208. A processor 208 may be a general-purpose processor or aspecial purpose processor (e.g., digital signal processors, applicationspecific integrated circuits, etc.). The one or more processors 208 canbe configured to execute computer-readable program instructions 212 thatare stored in the data storage 210 and are executable to provide thefunctionality of a UDV described herein.

The data storage 210 may include or take the form of one or morecomputer-readable storage media that can be read or accessed by at leastone processor 208. The one or more computer-readable storage media caninclude volatile and/or non-volatile storage components, such asoptical, magnetic, organic or other memory or disc storage, which can beintegrated in whole or in part with at least one of the one or moreprocessors 208. In some embodiments, the data storage 210 can beimplemented using a single physical device (e.g., one optical, magnetic,organic or other memory or disc storage unit), while in otherembodiments, the data storage 210 can be implemented using two or morephysical devices.

As noted, the data storage 210 can include computer-readable programinstructions 212 and perhaps additional data, such as diagnostic data ofthe UDV 200. As such, the data storage 210 may include programinstructions 212 to perform or facilitate some or all of the UDVfunctionality described herein. For instance, in the illustratedembodiment, program instructions 212 include a navigation module 214.

A. Sensors

In an illustrative embodiment, IMU 202 may include both an accelerometerand a gyroscope, which may be used together to determine an orientationof the UDV 200. In particular, the accelerometer can measure theorientation of the vehicle with respect to earth, while the gyroscopemeasures the rate of rotation around an axis. IMU 202 may take the formof or include a miniaturized MicroElectroMechanical System (MEMS) or aNanoElectroMechanical System (NEMS). Other types of IMUs may also beutilized.

An IMU 202 may include other sensors, in addition to accelerometers andgyroscopes, which may help to better determine position and/or help toincrease autonomy of the UDV 200. Two examples of such sensors aremagnetometers and pressure sensors. In some embodiments, a UDV mayinclude a low-power, digital 3-axis magnetometer, which can be used torealize an orientation independent electronic compass for accurateheading information. However, other types of magnetometers may beutilized as well. Other examples are also possible. Further, a UDV mayinclude some or all of the above-described inertia sensors as separatecomponents from an IMU.

An aerial implementation of UDV 200 may also include a pressure sensoror barometer, which can be used to determine the altitude of the UDV200. Alternatively, other sensors, such as sonic altimeters or radaraltimeters, can be used to provide an indication of altitude, which mayhelp to improve the accuracy of and/or prevent drift of an IMU.

In a further aspect, UDV 200 may include one or more sensors that allowthe UDV to sense objects in the environment. For instance, in theillustrated embodiment, UDV 200 includes ultrasonic sensor(s) 204.Ultrasonic sensor(s) 204 can determine the distance to an object bygenerating sound waves and determining the time interval betweentransmission of the wave and receiving the corresponding echo off anobject. A typical application of an ultrasonic sensor for unmannedvehicles is obstacle avoidance and, in the case of aerial vehicles,low-level altitude control. An ultrasonic sensor can also be used forvehicles that need to hover at a certain height or need to be capable ofdetecting obstacles. Other systems can be used to determine, sense thepresence of, and/or determine the distance to nearby objects, such as alight detection and ranging (LIDAR) system, laser detection and ranging(LADAR) system, and/or an infrared or forward-looking infrared (FLIR)system, among other possibilities.

In some embodiments, UDV 200 may also include one or more imagingsystem(s). For example, one or more still and/or video cameras may beutilized by UDV 200 to capture image data from the UDV's environment. Asa specific example, charge-coupled device (CCD) cameras or complementarymetal-oxide-semiconductor (CMOS) cameras can be used with unmannedvehicles. Such imaging sensor(s) have numerous possible applications,such as obstacle avoidance, localization techniques, ground tracking formore accurate navigation (e,g., by applying optical flow techniques toimages), video feedback, and/or image recognition and processing, amongother possibilities.

UDV 200 may also include a GPS receiver 206. The GPS receiver 206 may beconfigured to provide data that is typical of well-known GPS systems,such as the GPS coordinates of the UDV 200. Such GPS data may beutilized by the UDV 200 for various functions. As such, the UDV may useits GPS receiver 206 to help navigate to a delivery destination for anordered package, as indicated, at least in part, by GPS coordinatesprovided by a mobile device associated with a user placing the packageorder. Other examples are also possible.

B. Navigation and Location Determination

The navigation module 214 may provide functionality that allows the UDV200 to move about its environment and reach a desired location (e.g., adelivery destination). To do so, the navigation module 214 may controlthe mechanical features of the UDV. For example, in an aerial UDV,navigation module 214 may control the altitude and/or direction offlight by controlling the mechanical features of the UDV that affectflight (e.g., its rudder(s), elevator(s), aileron(s), and/or the speedof its propeller(s)). In a ground UDV, the navigation module may controlvehicle speed and direction by controlling the motors that drive thewheels or tracks and the actuators that control the vehicle steering.

In order to navigate the UDV 200 to a desired location, the navigationmodule 214 may implement various navigation techniques, such asmap-based navigation and localization-based navigation, for instance.With map-based navigation, the UDV 200 may be provided with a map of itsenvironment, which may then be used to navigate to a particular locationon the map. With localization-based navigation, the UDV 200 may becapable of navigating in an unknown environment using localization.Localization-based navigation may involve the UDV 200 building its ownmap of its environment and calculating its position within the mapand/or the position of objects in the environment. For example, as a UDV200 moves throughout its environment, the UDV 200 may continuously uselocalization to update its map of the environment. This continuousmapping process may be referred to as simultaneous localization andmapping (SLAM). Other navigation techniques may also be utilized.

In some embodiments, the navigation module 214 may navigate using atechnique that relies on waypoints. In particular, waypoints are sets ofcoordinates that identify points in physical space. For instance, anair-navigation waypoint may be defined by a certain latitude, longitude,and altitude. Accordingly, navigation module 214 may cause UDV 200 tomove from waypoint to waypoint, in order to ultimately travel to a finaldestination (e.g., a final waypoint in a sequence of waypoints).

In a further aspect, the navigation module 214 and/or other componentsand systems of the UDV 200 may be configured for “localization” to moreprecisely navigate to a target spot within a general destination. Morespecifically, it may be desirable in certain situations for a UDV to bewithin a threshold distance of the target spot in the generaldestination (e.g., a target drop-off spot within a delivery destination)where a payload 220 is being delivered by a UDV. To this end, a UDV mayuse a two-tiered approach in which it uses a more-generallocation-determination technique to navigate to the general destination(e.g., delivery destination) that is associated with the target spot(e.g., target drop-off spot for an object), and then use a more-refinedlocation-determination technique to identify and/or navigate to thetarget spot within the general destination.

For example, the UDV 200 may navigate to the general vicinity of adelivery destination where a payload 220 is being delivered usingwaypoints and/or map-based navigation. The UDV may then switch to a modein which it utilizes a localization process to locate and travel to atarget spot within the delivery destination. For instance, if the UDV200 is to deliver a payload to a user's home, the UDV 200 may need to besubstantially close to the target drop-off spot for the payload in orderto avoid delivery of the payload to undesired areas (e.g., onto a roof,into a pool, onto a neighbor's property, etc.). However, a GPS signalmay be limited to accurately guiding the UDV 200 to within a thresholddistance of the delivery destination (e.g., within a block of the user'shome) due to inherent accuracy limits of GPS systems. A more preciselocation-determination technique may then be used to find the specifictarget drop-off spot within the general vicinity of the deliverydestination.

Various types of location-determination techniques may be used toaccomplish localization of the target spot once the UDV 200 hasnavigated to the general vicinity of the delivery destination. Forinstance, the UDV 200 may be equipped with one or more sensory systems,such as, for example, ultrasonic sensors 204, infrared sensors (notshown), and/or other sensors, which may provide input that thenavigation module 214 utilizes to navigate autonomously orsemi-autonomously to the specific target spot.

As another example, once the UDV 200 reaches the general vicinity of thedelivery destination (or of a moving subject such as a person or theirmobile device), the UDV 200 may switch to a “fly-by-wire” mode where itis controlled, at least in part, by a remote operator, who can navigatethe UDV 200 to the target drop-off spot within the delivery destination.To this end, sensory data from the UDV 200 may be sent to the remoteoperator to assist them in navigating the UDV 200 to the target drop-offspot.

As yet another example, the UDV 200 may include a module that is able tosignal to a passer-by for assistance in reaching the specific targetdelivery location. For example, the UDV 200 may display a visual messagerequesting such assistance in a graphic display and/or play an audiomessage or tone through speakers to indicate the need for suchassistance, among other possibilities. Such a visual or audio messagemight indicate that assistance is needed in delivering the UDV 200 to aparticular person or a particular destination, and might provideinformation to assist the passer-by in delivering the UDV 200 to theperson or location (e.g., a description or picture of the person orlocation), among other possibilities. Such a feature can be useful in ascenario in which the UDV is unable to use sensory functions or anotherlocation-determination technique to reach the specific target spotwithin a delivery destination. However, this feature is not limited tosuch scenarios.

In some embodiments, once the UDV 200 arrives at the general vicinity ofa delivery destination, the UDV 200 may utilize a beacon from a user'sremote device (e.g., the user's mobile phone) to locate the person. Sucha beacon may take various forms. As an example, consider the scenariowhere a remote device, such as the mobile phone of a person whorequested a UDV delivery, is able to send out directional signals (e.g.,via an RF signal, a light signal and/or an audio signal). In thisscenario, the UDV 200 may be configured to navigate by “sourcing” suchdirectional signals—in other words, by determining where the signal isstrongest and navigating accordingly. As another example, a mobiledevice can emit a frequency, either in the human range or outside thehuman range, and the UDV 200 can listen for that frequency and navigateaccordingly. As a related example, if the UDV 200 is listening forspoken commands, then the UDV 200 could utilize spoken statements, suchas “I'm over here!” to source the specific location of the personrequesting delivery of a payload.

In an alternative arrangement, a navigation module may be implemented ata remote computing device, which communicates wirelessly with the UDV200. The remote computing device may receive data indicating theoperational state of the UDV 200, sensor data from the UDV 200 thatallows it to assess the environmental conditions being experienced bythe UDV 200, and/or location information for the UDV 200. Provided withsuch information, the remote computing device may determine altitudinaland/or directional adjustments that should be made by the UDV 200 and/ormay determine how the UDV 200 should adjust its mechanical features(e.g., its wheel(s), track(s), leg(s), rudder(s), elevator(s),aileron(s), and/or the speed of its propeller(s)) in order to effectuatesuch movements. The remote computing system may then communicate suchadjustments to the UDV 200 so it can move in the determined manner.

C. Communication System

In a further aspect, the UDV 200 includes one or more communicationsystems 216. The communications systems 216 may include one or morewireless interfaces and/or one or more wireline interfaces, which allowthe UDV 200 to communicate via one or more networks. Such wirelessinterfaces may provide for communication under one or more wirelesscommunication protocols, such as Bluetooth, WiFi (e.g., an IEEE 802.11protocol), Long-Term Evolution (LTE), WiMAX (e.g., an IEEE 802.16standard), a radio-frequency ID (RFID) protocol, near-fieldcommunication (NFC), and/or other wireless communication protocols. Suchwireline interfaces may include an Ethernet interface, a UniversalSerial Bus (USB) interface, or similar interface to communicate via awire, a twisted pair of wires, a coaxial cable, an optical link, afiber-optic link, or other physical connection to a wireline network.

In some embodiments, a UDV 200 may include communication systems 216that allow for both short-range communication and long-rangecommunication. For example, the UDV 200 may be configured forshort-range communications using Bluetooth and for long-rangecommunications under a CDMA protocol. In such an embodiment, the UDV 200may be configured to function as a “hotspot,” or in other words, as agateway or proxy between a remote support device and one or more datanetworks, such as a cellular network and/or the Internet. Configured assuch, the UDV 200 may facilitate data communications that the remotesupport device would otherwise be unable to perform by itself.

For example, the UDV 200 may provide a WiFi connection to a remotedevice, and serve as a proxy or gateway to a cellular service provider'sdata network, which the UDV might connect to under an LTE or a 3Gprotocol, for instance. The UDV 200 could also serve as a proxy orgateway to a high-altitude balloon network, a satellite network, or acombination of these networks, among others, which a remote device mightnot be able to otherwise access.

D. Power Systems

In a further aspect, the UDV 200 may include power system(s) 218. Thepower system 218 may include one or more batteries for providing powerto the UDV 200. In one example, the one or more batteries may berechargeable and each battery may be recharged via a wired connectionbetween the battery and a power supply and/or via a wireless chargingsystem, such as an inductive charging system that applies an externaltime-varying magnetic field to an internal battery.

E. Payloads

The UDV 200 may employ various systems and configurations in order totransport a payload 220. In some implementations, the payload 220 of agiven UDV 200 may include or take the form of a “package” designed totransport various goods to a target delivery location. For example, theUDV 200 can include a compartment, in which an item or items may betransported. Such a package may one or more food items, purchased goods,medical items, or any other object(s) having a size and weight suitableto be transported between two locations by the UDV. In otherembodiments, a payload 220 may simply be the one or more items that arebeing delivered (e.g., without any package housing the items).

In some embodiments, the payload 220 may be attached to the UDV andlocated substantially outside of the UDV during some or all of a travelpath by the UDV. For example, the package may be tethered or otherwisereleasably attached below an aerial UDV during flight to a targetlocation. In an embodiment where a package carries goods below theaerial UDV, the package may include various features that protect itscontents from the environment, reduce aerodynamic drag on the system,and prevent the contents of the package from shifting during aerial UDVflight.

For instance, when the payload 220 takes the form of a package fortransporting items, the package may include an outer shell constructedof water-resistant cardboard, plastic, or any other lightweight andwater-resistant material. Further, in order to reduce drag whentransported by an aerial UDV, the package may feature smooth surfaceswith a pointed front that reduces the frontal cross-sectional area.Further, the sides of the package may taper from a wide bottom to anarrow top, which allows the package to serve as a narrow pylon thatreduces interference effects on the wing(s) of the UDV. This may movesome of the frontal area and volume of the package away from the wing(s)of the UDV, thereby preventing the reduction of lift on the wing(s)cause by the package. Yet further, in some embodiments, the outer shellof the package may be constructed from a single sheet of material inorder to reduce air gaps or extra material, both of which may increasedrag on the system. Additionally or alternatively, the package mayinclude a stabilizer to dampen package flutter. This reduction influtter may allow the package to have a less rigid connection to the UDVand may cause the contents of the package to shift less during flight.

In order to deliver the payload, the aerial UDV may include aretractable delivery system that lowers the payload to the ground whilethe UDV hovers above. For instance, the UDV may include a tether that iscoupled to the payload by a release mechanism. A winch can unwind andwind the tether to lower and raise the release mechanism. The releasemechanism can be configured to secure the payload while being loweredfrom the UDV by the tether and release the payload upon reaching groundlevel. The release mechanism can then be retracted to the UDV by reelingin the tether using the winch.

In some implementations, the payload 220 may be passively released onceit is lowered to the ground. For example, a passive release mechanismmay include one or more swing arms adapted to retract into and extendfrom a housing. An extended swing arm may form a hook on which thepayload 220 may be attached. Upon lowering the release mechanism and thepayload 220 to the ground via a tether, a gravitational force as well asa downward inertial force on the release mechanism may cause the payload220 to detach from the hook allowing the release mechanism to be raisedupwards toward the aerial UDV. The release mechanism may further includea spring mechanism that biases the swing arm to retract into the housingwhen there are no other external forces on the swing arm. For instance,a spring may exert a force on the swing arm that pushes or pulls theswing arm toward the housing such that the swing arm retracts into thehousing once the weight of the payload 220 no longer forces the swingarm to extend from the housing. Retracting the swing arm into thehousing may reduce the likelihood of the release mechanism snagging thepayload 220 or other nearby objects when raising the release mechanismtoward the UDV upon delivery of the payload 220.

Active payload release mechanisms are also possible. For example,sensors such as a barometric pressure based altimeter and/oraccelerometers may help to detect the position of the release mechanism(and the payload) relative to the ground. Data from the sensors can becommunicated back to the aerial UDV and/or a control system over awireless link and used to help in determining when the release mechanismhas reached ground level (e.g., by detecting a measurement with theaccelerometer that is characteristic of ground impact). In otherexamples, the aerial UDV may determine that the payload has reached theground based on a weight sensor detecting a threshold low downward forceon the tether and/or based on a threshold low measurement of power drawnby the winch when lowering the payload.

Other systems and techniques for delivering a payload, in addition or inthe alternative to a tethered delivery system are also possible. Forexample, a UDV 200 could include an air-bag drop system or a parachutedrop system. Alternatively, a UDV 200 carrying a payload could simplyland on the ground at a delivery location. Other examples are alsopossible.

A ground-based UDV may include an articulated robotic arm that may beused to move the payload out of a storage compartment on the UDV andplace the object at a target drop-off spot within the deliverydestination.

IV. EXAMPLE UDV DEPLOYMENT SYSTEMS

UDV systems may be implemented in order to provide various UDV-relatedservices. In particular, UDVs may be provided at a number of differentlaunch sites that may be in communication with regional and/or centralcontrol systems. Such a distributed UDV system may allow UDVs to bequickly deployed to provide services across a large geographic area(e.g., that is much larger than the travel range of any single UDV). Forexample, UDVs capable of carrying payloads may be distributed at anumber of launch sites across a large geographic area (possibly eventhroughout an entire country, or even worldwide), in order to provideon-demand transport of various items to locations throughout thegeographic area. FIG. 3 is a simplified block diagram illustrating adistributed UDV system 300, according to an example embodiment.

In the illustrative UDV system 300, an access system 302 may allow forinteraction with, control of, and/or utilization of a network of UDVs304. In some embodiments, an access system 302 may be a computing systemthat allows for human-controlled dispatch of UDVs 304. As such, thecontrol system may include or otherwise provide a user interface throughwhich a user can access and/or control the UDVs 304.

In some embodiments, dispatch of the UDVs 304 may additionally oralternatively be accomplished via one or more automated processes. Forinstance, the access system 302 may dispatch one of the UDVs 304 totransport a payload to a target location, and the UDV may autonomouslynavigate to the target location by utilizing various on-board sensors,such as a GPS receiver and/or other various navigational sensors.

Further, the access system 302 may provide for remote operation of aUDV. For instance, the access system 302 may allow an operator tocontrol the flight of a UDV via its user interface. As a specificexample, an operator may use the access system 302 to dispatch a UDV 304to a delivery destination. The UDV 304 may then autonomously navigate tothe general vicinity of the delivery destination. At this point, theoperator may use the access system 302 to take control of the UDV 304and navigate the UDV to a target spot within the delivery destination(e.g., a user-designated target drop-off spot for an ordered package).Other examples of remote operation of a UDV are also possible.

In an illustrative embodiment, the UDVs 304 may take various forms. Forexample, each of the UDVs 304 may be a UDV such as those illustrated inFIGS. 1A-1E or 2. However, UDV system 300 may also utilize other typesof UDVs without departing from the scope of the invention. In someimplementations, all of the UDVs 304 may be of the same or a similarconfiguration. However, in other implementations, the UDVs 304 mayinclude a number of different types of UDVs. For instance, the UDVs 304may include a number of types of UDVs (e.g., ground UDVs, aerial UDVs,and water-based UDVs), with each type of UDV being configured for adifferent type or types of payload delivery capabilities.

The UDV system 300 may further include a remote device 306, which maytake various forms. Generally, the remote device 306 may be any devicethrough which a direct or indirect request to dispatch a UDV can bemade. (Note that an indirect request may involve any communication thatmay be responded to by dispatching a UDV, such as requesting a packagedelivery). In an example embodiment, the remote device 306 may be amobile phone, tablet computer, laptop computer, personal computer, orany network-connected computing device. Further, in some instances, theremote device 306 may not be a computing device. As an example, astandard telephone, which allows for communication via plain oldtelephone service (POTS), may serve as the remote device 306. Othertypes of remote devices are also possible.

Further, the remote device 306 may be configured to communicate withaccess system 302 via one or more types of communication network(s) 308.For example, the remote device 306 may communicate with the accesssystem 302 (or a human operator of the access system 302) bycommunicating over a POTS network, a cellular network, and/or a datanetwork such as the Internet. Other types of networks may also beutilized.

In some embodiments, the remote device 306 may be configured to allow auser to request delivery of one or more items to a desired location. Forexample, a user could request UDV delivery of a package to their homevia their mobile phone, tablet, or laptop. As another example, a usercould request dynamic delivery to wherever they are located at the timeof delivery. To provide such dynamic delivery, the UDV system 300 mayreceive location information (e.g., GPS coordinates, etc.) from theuser's mobile phone, or any other device on the user's person, such thata UDV can navigate to the user's location (as indicated by their mobilephone).

In an illustrative arrangement, the central dispatch system 310 may be aserver or group of servers, which is configured to receive dispatchmessages requests and/or dispatch instructions from the access system302. Such dispatch messages may request or instruct the central dispatchsystem 310 to coordinate the deployment of UDVs to various targetlocations. The central dispatch system 310 may be further configured toroute such requests or instructions to one or more local dispatchsystems 312. To provide such functionality, the central dispatch system310 may communicate with the access system 302 via a data network, suchas the Internet or a private network that is established forcommunications between access systems and automated dispatch systems.

In the illustrated configuration, the central dispatch system 310 may beconfigured to coordinate the dispatch of UDVs 304 from a number ofdifferent local dispatch systems 312. As such, the central dispatchsystem 310 may keep track of which UDVs 304 are located at which localdispatch systems 312, which UDVs 304 are currently available fordeployment, and/or which services or operations each of the UDVs 304 isconfigured for (in the event that a UDV fleet includes multiple types ofUDVs configured for different services and/or operations). Additionallyor alternatively, each local dispatch system 312 may be configured totrack which of its associated UDVs 304 are currently available fordeployment and/or are currently in the midst of item transport.

In some cases, when the central dispatch system 310 receives a requestfor UDV-related service (e.g., transport of an item) from the accesssystem 302, the central dispatch system 310 may select a specific UDV304 to dispatch. The central dispatch system 310 may accordinglyinstruct the local dispatch system 312 that is associated with theselected UDV to dispatch the selected UDV. The local dispatch system 312may then operate its associated deployment system 314 to launch theselected UDV. In other cases, the central dispatch system 310 mayforward a request for a UDV-related service to a local dispatch system312 that is near the location where the support is requested and leavethe selection of a particular UDV 304 to the local dispatch system 312.

In an example configuration, the local dispatch system 312 may beimplemented as a computing system at the same location as the deploymentsystem(s) 314 that it controls. For example, the local dispatch system312 may be implemented by a computing system installed at a building,such as a warehouse, where the deployment system(s) 314 and UDV(s) 304that are associated with the particular local dispatch system 312 arealso located. In other embodiments, the local dispatch system 312 may beimplemented at a location that is remote to its associated deploymentsystem(s) 314 and UDV(s) 304.

Numerous variations on and alternatives to the illustrated configurationof the UDV system 300 are possible. For example, in some embodiments, auser of the remote device 306 could request delivery of a packagedirectly from the central dispatch system 310. To do so, an applicationmay be implemented on the remote device 306 that allows the user toprovide information regarding a requested delivery, and generate andsend a data message to request that the UDV system 300 provide thedelivery. In such an embodiment, the central dispatch system 310 mayinclude automated functionality to handle requests that are generated bysuch an application, evaluate such requests, and, if appropriate,coordinate with an appropriate local dispatch system 312 to deploy aUDV.

Further, some or all of the functionality that is attributed herein tothe central dispatch system 310, the local dispatch system(s) 312, theaccess system 302, and/or the deployment system(s) 314 may be combinedin a single system, implemented in a more complex system, and/orredistributed among the central dispatch system 310, the local dispatchsystem(s) 312, the access system 302, and/or the deployment system(s)314 in various ways.

Yet further, while each local dispatch system 312 is shown as having twoassociated deployment systems 314, a given local dispatch system 312 mayalternatively have more or fewer associated deployment systems 314.Similarly, while the central dispatch system 310 is shown as being incommunication with two local dispatch systems 312, the central dispatchsystem 310 may alternatively be in communication with more or fewerlocal dispatch systems 312.

In a further aspect, the deployment systems 314 may take various forms.In general, the deployment systems 314 may take the form of or includesystems for physically launching one or more of the UDVs 304. Suchlaunch systems may include features that provide for an automated UDVlaunch and/or features that allow for a human-assisted UDV launch.Further, the deployment systems 314 may each be configured to launch oneparticular UDV 304, or to launch multiple UDVs 304.

The deployment systems 314 may further be configured to provideadditional functions, including for example, diagnostic-relatedfunctions such as verifying system functionality of the UDV, verifyingfunctionality of devices that are housed within a UDV (e.g., a payloaddelivery apparatus), and/or maintaining devices or other items that arehoused in the UDV (e.g., by monitoring a status of a payload such as itstemperature, weight, etc.).

In some embodiments, the deployment systems 314 and their correspondingUDVs 304 (and possibly associated local dispatch systems 312) may bestrategically distributed throughout an area such as a city. Forexample, the deployment systems 314 may be strategically distributedsuch that each deployment system 314 is proximate to one or more payloadpickup locations (e.g., near a restaurant, store, or warehouse).However, the deployment systems 314 (and possibly the local dispatchsystems 312) may be distributed in other ways, depending upon theparticular implementation. As an additional example, kiosks that allowusers to transport packages via UDVs may be installed in variouslocations. Such kiosks may include UDV launch systems, and may allow auser to provide their package for loading onto a UDV and pay for UDVshipping services, among other possibilities. Other examples are alsopossible.

In a further aspect, the UDV system 300 may include or have access to auser-account database 316. The user-account database 316 may includedata for a number of user accounts, and which are each associated withone or more person. For a given user account, the user-account database316 may include data related to or useful in providing UDV-relatedservices. Typically, the user data associated with each user account isoptionally provided by an associated user and/or is collected with theassociated user's permission.

Further, in some embodiments, a person may be required to register for auser account with the UDV system 300, if they wish to be provided withUDV-related services by the UDVs 304 from UDV system 300. As such, theuser-account database 316 may include authorization information for agiven user account (e.g., a username and password), and/or otherinformation that may be used to authorize access to a user account.

In some embodiments, a person may associate one or more of their deviceswith their user account, such that they can access the services of UDVsystem 300. For example, when a person uses an associated mobile phoneto, e.g., place a call to an operator of the access system 302 or send amessage requesting a UDV-related service to a dispatch system, the phonemay be identified via a unique device identification number, and thecall or message may then be attributed to the associated user account.Other examples are also possible.

V. EXAMPLE OBJECT DELIVERY OPERATIONS

FIG. 4 illustrates a flow diagram 400 of operations that may beperformed by one or more computing devices within a delivery system(e.g., UDV system) to deliver an object (e.g., package) to a targetdrop-off spot within a delivery destination. In one example, theoperations of flow diagram 400 may be performed by a server device incommunication with a delivery vehicle and a client computing device.Alternatively, a control system of the delivery vehicle may beconfigured to perform the operations of flow diagram 400. In someembodiments, the operations of flow diagram 400 may be distributedbetween one or more server devices, one or more delivery vehicles, andone or more client computing devices. The manner in which the operationsare divided may be based on a communication bandwidth associated withthe operations herein described.

Within examples, a delivery destination may be a geographic location towhich delivery of an object is requested. The delivery destination maybe, for example, a residence of a user requesting delivery of theobject. In other examples, the delivery destination may be the user'swork, a residence belonging to the user's relatives, a current locationof the user, or a future location of the user, among otherpossibilities. Additionally, within examples, a target drop-off spot maybe an area within the delivery destination at which a delivery vehicleis requested and/or expected to place the object upon delivery of theobject to the delivery destination. The area of the target drop-off spotmay be a subset of the area of the delivery destination. The targetdrop-off spot may be designated by the user to have the package placedin a spot that the user regards as safe and/or convenient.

In block 402, an indication of a target drop-off spot for an object maybe received from a client computing device. The indication may representa spot within a first virtual model of a first region of a deliverydestination. The first virtual model may represent first physicalfeatures of the first region of the delivery destination. Withinexamples, the physical features may include topological features,vegetation, buildings, architectural and/or structural features ofbuildings, and other objects contained in, adjacent to, or observablefrom a region of the delivery destination. The object may be containedin a box or other package to be delivered by an autonomous deliveryvehicle to the delivery destination.

The first virtual model may be generated based on sensor data receivedfrom one or more sensors on or connected to the client computing device.For example, a user may capture a plurality of images of the firstregion of the delivery destination using a camera on the clientcomputing device. Each image may be associated with additional sensordata from one or more additional sensors on the client computing devicesuch as, for example, GPS module, magnetometers, gyroscopes, andaccelerometers. The client computing device may transmit the images,along with the additional sensor data, to a remote server that maygenerate the first virtual model based on the images and the additionalsensor data. Alternatively, the client computing device may beconfigured to generate the first virtual model locally.

The first virtual model may then be displayed by way of a user interfaceof the client computing device. The client computing device may promptfor designation of a target drop-off spot within the first virtualmodel. User designation of the target drop-off spot may be received byway of the user interface. Specifically, the user may use gestures(e.g., touching, clicking, scrolling, etc.) to place a virtual“bullseye” marker within the displayed virtual model to designate thetarget drop-off spot. Thus, a user may select a spot that the user deemsappropriate or safe for package delivery. For example, a user maydesignate that packages be placed next to the front door of the user'sresidence in a spot where the packages will not block the front door.Similarly, the user may designate to have packages placed in a backyardof the user's residence, behind a locked gate, as opposed to in thefront of the user's residence.

Once indication of the target drop-off spot for the object is received,an unmanned delivery vehicle may be dispatched to deliver the object tothe delivery location. The delivery vehicle may navigate to within athreshold distance (e.g., within several meters) of the deliverydestination based on GPS coordinates of the delivery destination. Inorder to locate the delivery destination and, more specifically, thetarget drop-off spot within the delivery destination, the deliveryvehicle may rely on additional, non-GPS sensor data after coming withinthe threshold distance of the delivery destination.

Accordingly, in block 404, sensor data may be received from one or moresensors on the delivery vehicle. The sensor data may be indicative of asecond region of the delivery destination. The sensors may include acamera, a stereo camera, a depth sensor, LIDAR, accelerometers,magnetometers, and gyroscopes, among other possibilities. In someembodiments, the delivery vehicle may include all of the sensorsincluded on the client computing device from which sensor data wascaptured to generate the first virtual model. Accordingly, sensorreadings across multiple sensors of the delivery vehicle may be comparedto sensor readings from sensors of the same types on the clientcomputing device. Comparing sensor data across multiple sensors mayallow for a more efficient scanning of the delivery destination. Forexample, based on the magnetometer readings, the delivery vehicle may,while scanning the delivery destination, orient itself in the samedirection as the client computing device at the time of data acquisitionfor the first virtual model, thus increasing the probability of scanningthe same region of the delivery destination.

In block 406, a second virtual model of the second region of thedelivery destination may be determined based on the sensor data. Thesecond virtual model may represent second physical features of thesecond region of the delivery destination. As with the first virtualmodel, the physical features represented by the second virtual model mayinclude topological features, vegetation, buildings, and other objectscontained in, adjacent to, or observable from the second region of thedelivery destination. The second virtual model, as well as the firstvirtual model, may be a three dimensional (3D) model such as a pointcloud, wireframe model, surface model, or solid model. Alternatively,the model may be two dimensional and may include depth information.Further, in some embodiments, the model may be a collection of multipleimages representing the delivery destination.

In block 408, a mapping may be determined between one or more of thefirst physical features and one or more of the second physical features.In particular, the mapping may be between physical features representedin both virtual models. The mapping may be used to determine anoverlapping region between the first virtual model and the secondvirtual model. In other words, the mapping may spatially synchronize thedelivery vehicle's perception of the environment with the first virtualmodel to locate the target drop-off spot within the second virtualmodel. Determining the mapping may include determining a geometrictransformation between the one or more of the first physical featuresand the one or more of the second physical features. In someembodiments, the operations of blocks 404, 406, and 408 may be iterateduntil an overlapping region is found. Specifically, the second virtualmodel may be expanded based on additional sensor data until a mappingwith a sufficiently high confidence level is found between the first andsecond virtual models.

In block 410, a position of the target drop-off spot within the secondvirtual model may be determined based on the overlapping region. Inparticular, if the region of the delivery destination containing thetarget drop-off spot is not already represented by the second virtualmodel, the delivery vehicle may acquire additional sensor datarepresenting the region of the delivery destination containing thetarget drop-off spot. The determined geometric transformation may beapplied to coordinates of the target drop-off spot within the firstvirtual model to determine coordinates of the target drop-off spotwithin the second virtual model. Thus, a position of the target drop-offspot may be determined within the delivery vehicle's perception of thedelivery destination.

In block 412, based on the position of the target drop-off spot withinthe second virtual model, instructions may be provided to navigate thedelivery vehicle to the target drop-off spot to place the object at thetarget drop-off spot. In one example, machine learning algorithms may beutilized to plan the path through which to navigate the vehicle to thetarget drop-off spot to avoid obstacles, hazards, and/or otherenvironmental features within the delivery destination.

VI. EXAMPLE TARGET DROP-OFF SPOT DESIGNATION OPERATIONS

FIGS. 5A, 5B, and 5C illustrate example implementations for generating afirst virtual model of a first region of a delivery destination. Thefirst virtual model may be generated based on sensor data from a clientcomputing device used to order a package and request delivery of thepackage to a target drop-off spot within the delivery destination. Thefirst virtual model may be generated to allow the target drop-off spotto be designated within the delivery destination by way of a userinterface of the client computing device. The first virtual model mayadditionally serve as a point of reference for the delivery vehicle.Namely, the delivery vehicle may compare a second virtual model,generated based on sensor data captured by the delivery vehicle, to thefirst virtual model to localize the delivery vehicle with respect to thetarget drop-off spot.

In one example, client computing device 500 (e.g., a smartphone) mayprompt, by way of a user interface, to take a picture of a region of thedelivery destination (e.g., front yard of a home, backyard, balcony,camping ground, etc.) where the user desires to have one or morepackages delivered and dropped off, as shown in FIG. 5A. Computingdevice 500 may further prompt for additional images to be taken of thedelivery destination, as shown in FIGS. 5B and 5C, based on which a morecomplete model may be generated. In response to the prompts, a user maypan the camera of the computing device to capture additional images ofthe region within the delivery destination where the user desires tohave the packages dropped off. In some embodiments, the client computingdevice may capture a continuous video instead of discrete images.Accordingly, the computing device may prompt to pan the camera over thedesired region of the delivery destination.

As successive images are captured, either discretely or from a videostream, the images may be used to construct the first virtual model ofthe first region of the delivery destination represented by the images.In some embodiments, the images may be transmitted to a remote computingdevice configured to generate the virtual model from the images.Alternatively, the client computing device may be configured to generatethe model from the images locally.

Each image may be associated with a plurality of additional sensor dataacquired by the client computing device at the time of image capture,such as magnetometer, gyroscope, and accelerometer readings that may aidin determining a position and orientation from which the respectiveimage was captured. The images, along with the corresponding additionalsensor data, may be used to generate a model of the region of thedelivery destination represented by the images. The model may be threedimensional (3D) such as, for example, a point cloud model, a wireframemodel, a shell model, or a solid model, among other possibilities.Alternatively, in some embodiments, the model may be two dimensional(2D) and may include a depth map. Further, the model may be a collectionof multiple images, each associated with corresponding additional sensordata. Yet further, in some embodiments, the model may include all of theabove mentioned model types. Accordingly, the effects of errors andartifacts of each modeling technique on the mapping determined betweenthe first and second virtual models may be reduced or minimized byfinding multiple mappings, each based on a different model type.

While the model is being constructed, the client computing device mayprompt the user to pan the camera over a portion of the first region ofthe delivery destination for which additional sensor data may be neededto construct an accurate model. Alternatively or additionally, thecomputing device may prompt the user to walk to a different location toacquire sensor data from a different perspective, position, and/ororientation. The prompts make take the form of text and/or arrowsindicating in which direction to move or pan the client computing deviceto acquire the additional sensor data. In some embodiments, the clientcomputing device may display a real-time visualization or simulation ofthe model generation process. For example, as the virtual model is builtup, physical features displayed on the user interface (e.g., in thefull-screen viewfinder of the client computing device) may be replacedby virtual representations of the respective features to provide theuser with feedback. Based on the feedback, the user may pan the cameraover portions of the environment that have not been replaced withvirtual representations.

When the first virtual model 600 of the first region of the deliverydestination is complete, the model may be displayed by way of the userinterface of the client computing device 500, as illustrated in FIG. 6.The first virtual model 600 may include virtual representations of aplurality of physical features within the first region of the deliverydestination (e.g., delivery destination house 602, neighboring houses601 and 604, windows 606, 608, 610, 614, 626, 628, and 632, doors 612and 630, vegetation 620 and 622, gate 618, stairs 616 and 634, walkway624, and sidewalk 636). In some embodiments, the virtual model mayrepresent the physical features as point clouds, wireframes, surfaces,and/or solids. Alternatively, the underlying virtual structures may behidden from the user (but may nevertheless form parts of the model).Instead, the computing device might display one or more of the capturedimages to provide the user with a more intuitive representation of themodel.

The client computing device may prompt for designation of a targetdrop-off spot for an object within the delivery destination. A user maybe able to indicate a desired target drop-off spot by tapping on aregion of the displayed virtual model 600 to place a virtual “bullseye”marker 640 at that location. For example, the user may tap in region 638to place bullseye 640 in region 638. Upon delivery, the requested objectwill be placed by the delivery vehicle within the center of bullseye640.

In some embodiments, the virtual model 600 may be navigable by way ofinput through the user interface to allow the perspective or orientationfrom which the model is displayed to be changed. Navigation through thevirtual model 600 may allow the target drop-off spot to be selected withgreater accuracy. In some examples, the size of bullseye 640 may beproportional to a size of the object ordered for delivery and/or mayresemble a shape of the object (e.g., the bullseye may be rectangularwhen the requested object is packed in a rectangular box). The dynamicbullseye size and shape may aid in determining whether a designatedtarget drop-off spot within the delivery destination is suitable toreceive the requested object.

Additionally or alternatively, once the bullseye target is designatedwithin the virtual model, the system may determine whether thedesignated target drop-off spot is appropriate for the size of objectrequested as well as whether the delivery vehicle will be able to reachthe target drop-off spot. For example, a ground-based delivery vehiclemight be unable to deliver to a target drop-off spot that can only bereached by traversing stairs. An aerial vehicle might be unable todeliver to a target drop-off spot that does not provide sufficientaerial clearance.

In some instances, the user may omit capturing images to determine avirtual model of a portion of the delivery destination. In particular,virtual models may already be available of portions of the deliverydestination to which the user is requesting delivery of the object.Thus, a computing device may display the available virtual models toallow for designation of a target drop-off spot within a virtual modelor selection of a target drop-off spot from one or more previouslydesignated target drop-off spots. Further, in some embodiments, theclient computing device may prompt the user to designate one or moreadditional “backup” target drop-off locations. The package may be placedin the backup target drop-off location when the primary target drop-offlocation is already occupied by a package or obstacle or when theprimary target drop-off spot cannot be located or reached by thedelivery vehicle.

VII. EXAMPLE DELIVERY VEHICLE GUIDANCE OPERATIONS

In response to a request for object/package delivery, a delivery servicemay dispatch (e.g., by way of UDV system 300) a delivery vehicle withthe requested object to the delivery destination. The delivery vehiclemay navigate from its current location (e.g., a warehouse storing therequested object) to the delivery destination based primarily on GPSinformation.

In planning the path or trajectory to the delivery destination, thecontrol system of the delivery vehicle may consider informationrepresented by the first virtual model. For example, the control systemmay determine that the client computing device was pointed north whensensor data used to generate the first virtual model was captured.Accordingly, in order to increase or maximize the probability oflocating and observing, by sensors on the delivery vehicle, the regionrepresented by the first virtual model, the delivery vehicle mayapproach the delivery destination from the south, moving in a northerlydirection. In some instances, the northerly approach may naturallyresult from the relative geographic location between the originationlocation of the object and the delivery destination (e.g., deliverydestination lies north of the origination location). Alternatively, thepath or trajectory may be planned to navigate the vehicle past thedelivery destination (e.g., south of the delivery destination) andsubsequently approach the delivery destination from the desireddirection (e.g., northerly approach).

Upon arriving within a threshold distance of the delivery destination(e.g., within a horizontal and/or vertical accuracy limit of the GPSsystem), the control system may begin capturing data from sensors on thedelivery vehicle for localization and navigation of the vehicle to thetarget drop-off spot for the object, as illustrated in FIG. 7A. Inparticular, FIG. 7A illustrates aerial UDV 180 capturing sensor datarepresenting a portion of delivery destination 602. Aerial UDV 180 mayhover about, pan the camera on the UDV, or a combination thereof tosweep the sensor field of view 702 over the delivery destination toacquire sensor data representing a region of the delivery destination(e.g., the front of the house 602). The sensor data from the deliveryvehicle may be captured from a different position and/or a differentorientation than that of the client computing device used to generatethe first virtual model. For example, aerial UDV 180 may capture thesensor data from a greater height than the client computing device whilea ground UDV may capture the sensor data from a lesser height than theclient computing device. The horizontal perspective of the sensor datacaptured by the delivery vehicle may equally differ from that of thefirst virtual model.

Once sensor data is captured, a second virtual model 700 may begenerated based on the sensor data, as illustrated in FIG. 7B. Thesecond virtual model may reflect the perspective from which the sensordata has been captured. In particular, due to the different perspective,the second virtual model may include representations of windows 642,644, and 646 that are not included in the first virtual model.Additionally, the second virtual model may lack, due to the differentperspective, representations of windows 606, 608, 626, 628, and 632,door 630, and stairs 634. In some embodiments, the model may includeperspective distortion. Namely, the perspective from which the sensordata is captured may be reflected in the shapes and relative sizes ofthe various physical features represented in the virtual model (i.e.,physical features closer to the camera may appear larger than physicalfeatures further away from the camera). The type of model used torepresent the delivery destination (e.g., 3D vs. 2D with depth vs.simple images) may additionally influence the extent to which thevirtual representations of the physical features reflect the perspectivefrom which the physical objects have been observed. For example, a 3Dmodel might be free of perspective distortion.

A mapping may be determined between the first model 600 and the secondmodel 700 to determine an overlapping region between the first andsecond models, as illustrated in FIG. 8A. In some example embodiments,the mapping may be determined by directly comparing model 700 to model600. In particular, the system may determine a mapping between one ormore of first physical features represented in the first virtual model600 and one or more of second physical features represented in thesecond virtual model 700. Determining the mapping may be carried out intwo steps. First, the system may determine physical features that arecommon between the first virtual model and the second virtual model. Forexample, house 601, house 602, house 604, vegetation 620 and 622,windows 610 and 614, and door 612 are represented in both the firstmodel 600 and the second model 700. In contrast, windows 606 and 608 arerepresented only in model 600 while windows 644 and 646 are representedonly in model 700. Thus, some physical features might be represented inonly one of the models and a mapping between these features might not bepossible.

Determining the mapping may further include determining a geometrictransformation between features represented in the first model andfeatures represented in the second model to establish a spatialrelationship between the two models. For example, the geometrictransformation may be represented as a matrix. In some embodiments, onetransformation may be sufficient to map representations of the physicalfeatures between the two models. Alternatively or additionally, eachrespective physical feature may be associated with an individualtransformation that maps the representation of the respective feature inthe first model to the representation of the respective feature in thesecond model.

In FIG. 8A, lines 804, 806, 808, 810, 812, 814, and 816 indicate examplemappings of representations of physical objects between the firstvirtual model 600 and the second virtual model 700. Specifically, line804 represents a mapping between representations of roof corner ofneighboring house 601, line 806 represents a mapping betweenrepresentations of vegetation 620, line 808 represents a mapping betweenrepresentations of window 610, line 810 represents a mapping betweenrepresentations of door 612, line 812 represents a mapping betweenrepresentations of roof peak of the house 602 (e.g., house associatedwith or representing the delivery destination), line 814 represents amapping between representations of vegetation 622, and line 816represents a mapping between representations of roof corner ofneighboring house 604. Each of lines 804-816 may be associated with anindividual transformation (e.g., a transformation matrix) specific tothe respective feature indicating the mapping. Alternatively oradditionally, the mapping may be represented by a single globaltransformation between models 600 and 700.

The mapping between two models may be carried out using algorithms suchas, for example, Random Sample Consensus (RANSAC), Least Median Squaresestimation, Iterative Closest Point, Robust Point Matching, KernelCorrelation, and/or Coherent Point Drift, among other possibilities.Additionally or alternatively, the mapping may utilize machine learningalgorithms to improve the mapping accuracy and reliability. For example,a decision tree, support vector machine, or Bayesian algorithm may beused to group representations of physical features within a virtualmodel into categories (e.g., vegetation, doors, windows, stairs,sidewalks, roadways, etc.). Accordingly, the system may only try to mapa particular feature within the second model to features in the firstmodel having the same class (e.g., try to map a window to other windowsbut not to doors or vegetation). In determining the mapping, the systemmay account for variations caused by time of day, weather, seasons(e.g., leaves or snow), changes in appearance of structures in thedelivery location (e.g., house painted a different color since the firstmodel was generated) as well as additional or missing objects at thedelivery location (e.g., fallen branches, rearranged patio furniture,etc.).

The mapping may be used to determine an overlapping region between thefirst and second models. The overlap may be a geometric intersectionbetween a portion of the second virtual model and a correspondingportion of the first virtual model. In particular, the extent to whichthe second representations of physical features in second model 700correspond or map to representations of first physical features in firstmodel 600 may determine whether second model 700 and first model 600represent any common regions of the delivery destination.

For example, if the delivery vehicle initially scans a region adjacentto the wrong house due to inaccuracies in vehicle positioning based onGPS data, the system may determine that the mapping between the firstmodel and the second model is weak or cannot be determined. Inparticular, a first portion of the second virtual model may bedetermined based on the initial sensor data. The first portion mayrepresent physical features of a first sub-region of the second regionof the delivery destination. The system may determine that the firstportion of the second virtual model does not overlap with the firstvirtual model by determining that a confidence level of a mappingbetween the physical features represented in the first portion of thesecond virtual model and the first physical features is below athreshold confidence value. In other words, physical features within thefirst model do not correspond spatially to physical features within thefirst portion of the second model (e.g., the neighboring house may havea door and windows but the spatial relationship between the doors andwindows does not match that of the first virtual model).

In response to determining that the confidence level of the mapping isbelow the threshold confidence value, the delivery vehicle may collectadditional sensor data indicative of a second-sub region of the secondregion of the delivery destination. A second portion of the secondvirtual model may be determined based on the additional sensor data. Thesecond portion may represent physical features of the second sub-regionof the second region of the delivery destination. A mapping may bedetermined between one or more of the first physical features and one ormore of the physical features of the second sub-region to determine theoverlapping region between the first virtual model and the secondportion of the second virtual model. Additionally, the system maydetermine that a confidence level of the mapping between the one or moreof the first physical features and the one or more of the physicalfeatures of the second sub-region is greater than the thresholdconfidence value.

In some instances, an overlapping region may be determined between thefirst virtual model and the second virtual model. However, theoverlapping region might not contain the target drop-off spot.Accordingly, determining the position of the target drop-off spot withinthe second virtual model may include determining, based on theoverlapping region, a path through which to move the delivery vehicle tocollect additional sensor data indicative of a region of the deliverydestination containing the target drop-off spot. In response,instructions may be provided to navigate the delivery vehicle along thedetermined path and collect the additional sensor data while navigatingalong the determined path. Further, the second virtual model may beupdated based on the additional sensor data to represent the targetdrop-off spot and physical features of the region of the deliverydestination containing the target drop-off spot.

The operations of building-up the second virtual model and determining amapping between the first and second models may be repeated until thedelivery vehicle identifies a region of the delivery destination thatmatches the region represented by the first virtual model (i.e., untilthe overlapping region is identified). In other words, the deliveryvehicle may search the general vicinity of the delivery destinationuntil the region specified by the client computing device is found. Themapping operations may provide an objective metric for determiningwhether the region specified by the package recipient has been located.The operations of sensor data collection, model generation, and mappingmay proceed sequentially or in parallel and may be carried out locally,by a control system of the delivery vehicle, or remotely, by a serverdevice.

Once an overlapping region is identified, the overlapping region may beused to determine a position of the target drop-off spot 818 in thesecond virtual model 700. Specifically, determining the mapping mayinclude determining a geometric transformation between representationsof the one or more of the first physical features and representations ofthe one or more of the second physical features. Determining theposition of target drop-off spot 818 in the second virtual model 700 mayinclude applying the determined geometric transform to coordinates ofthe target drop-off spot within the first virtual model to determinecoordinates of the target drop-off spot within the second virtual model.By determining the position of the target drop-off spot within thesecond virtual model, the control system may determine a spatialrelationship between the delivery vehicle and the target drop-off spotwithin the delivery vehicle's perception of the delivery destination(e.g., within the delivery vehicle's coordinate system).

Accordingly, the delivery vehicle may navigate to the target drop-offspot to place the object at the target drop-off spot based on thedetermined spatial relationship between the delivery vehicle and thetarget drop-off location, as illustrated in FIG. 8B. In particular, FIG.8B illustrates aerial UDV 180 hovering above target drop-off spot 818before placing package 820 down at the target drop-off spot 820. Thus,package 820 is placed in the target drop-off spot specified by thepackage recipient.

In some examples, after placing the object at the target drop-off spot,the delivery vehicle may capture image data (e.g., photo or video)showing the object successfully disposed at the target drop-off spot.The image data representing the object disposed at the target drop-offspot may be received and stored by a remote server within the deliverysystem as proof of delivery. A delivery confirmation may be transmittedto the client computing device. The delivery confirmation may includethe image data representing the object disposed at the target drop-offspot. Further, in some embodiments, the delivery vehicle mayadditionally capture a video while the delivery vehicle is placing theobject at the target drop-off spot. The video may be transmitted to andstored by the remote server and may be included in the deliveryconfirmation. In some examples, the video may be streamed live to theclient computing device while the delivery vehicle is placing the objectat the target drop-off spot. The client computing device may be used toaid in navigation of the delivery vehicle by tapping or otherwiseindicating regions of the delivery destination to which the vehicleshould navigate to find the target drop-off spot.

In some embodiments, the delivery vehicle may include the same sensorsas the client computing device used to gather the sensor data based onwhich the first virtual model is generated. For example, the clientcomputing device and the vehicle may each include a camera, gyroscopes,accelerometers, magnetometers, and a GPS unit, among otherpossibilities. Accordingly, the first virtual model and the secondvirtual model (i.e., model generated based on sensor data from thedelivery vehicle) may each be based on sensor data from the same sensortypes. Thus, when the system searches for overlap between the first andsecond models, the system may consider not only the determined modelsbut also the underlying raw data from which the models were generated(e.g., the direction from which the image data was captured).

Notably, the operations herein described do not rely on any landingbeacons, fiducial markers, or other physical markers that serve aslocalization points for the delivery vehicle. Instead, navigation of thedelivery vehicle to the target drop-off spot relies on detection offeatures already present within the environment of the deliverydestination. Thus, package delivery may easily be requested to locationsnot equipped with landing beacons or specialized landing markers. Forexample, a use may request and receive a package while at a locationother than the user's home such as a park or camping ground. Inparticular, the systems and operations herein described may extendpackage delivery services to geographic regions not currently served byexisting package delivery services due to lack of roads and/or organizedstreet addresses, among other reasons.

VIII. ADDITIONAL EXAMPLE MAPPING OPERATIONS

In some example embodiments, the mapping may be determined by comparingmodels 600 and 700, shown in FIGS. 6 and 7B, respectively, to apredetermined master model representing the geographical region of thedelivery destination. FIG. 9 illustrates a portion of an example mastermodel 900 representing the geographical region surrounding deliverydestination 602 and including houses 601 and 604 as well as houses 901,902, 904, and 906 that are not included in model 600. The master model900 may be determined at a time prior to the delivery request for theobject. For example, the master model 900 may be determined based onsensor data from one or more delivery vehicles operating in thegeographical region of the delivery destination 602 while en route toother delivery destinations. In another example, master model 900 may bedetermined and updated based on previously-determined virtual modelsgenerated during prior object deliveries to locations within thegeographic area and stored in a database (e.g., deliveries to houses901, 902, 904, and 906. In a further example, master model 900 may begenerated and updated based on image data from a mapping service thatincludes street-level images of geographic locations.

Master model 900 may be used to locate the delivery destination 602 andto localize the delivery vehicle within the general geographic vicinityof the delivery destination 602. In particular, after the first model600 is generated, it may be compared or mapped to master model 900 tolocate the first model 600 within master model 900. Additionally, themapping of model 600 to master model 900 may be used to determine aposition of the target drop-off spot 640 within master model 900.

Upon arriving within the general vicinity of delivery destination 602,the delivery vehicle may begin collecting sensor data based on which thesecond virtual model 700 may be generated. As the second model 700 isgenerated and updated, model 700 or a portion thereof may be compared ormapped to the master model 900 to determine the position and orientationof the delivery vehicle within the general geographic location ofdelivery destination 602 represented by model 900. The relativegeographic positioning between models 600 and 700 may be determinedbased on their respective positions within master model 900. Thus, insome examples, models 600 and 700 might not be compared directly to oneanother. Instead, master model 900 may serve as an intermediary indetermining the mapping between models 600 and 700.

The control system of the delivery vehicle may use master model 900 todetermine a direction in which to move the delivery vehicle to find andscan delivery destination 602 and locate the target drop-off spot 640.Notably, by using master model 900, the control system may more quicklyand accurately determine the direction in which to move to locatedelivery destination 602 than if the control system had to estimate orpredict the relative positioning of models 600 and 700 without mastermodel 900. Model 900 may be particularly useful when the initialportions of model 700 do not overlap with model 600.

The delivery vehicle may be localized with respect to the deliverydestination 602 based on distinct patterns of physical features withinmaster model 900. For example, house 901 may include trees 914 and 916planted to the left thereof, house 902/904 may form a U-shape, and house906 may include a large fenced yard 908, tree 910, and shrubs 912 and918. When, upon arrival within the general geographic vicinity ofdelivery destination 602, the delivery vehicle first collects sensordata indicative of house 906, the system may determine that to reachdelivery destination 602, the delivery vehicle should proceed furthernorth (i.e., up, along the page). Alternatively, when, upon arrivalwithin the general geographic vicinity of delivery destination 602, thedelivery vehicle first collects sensor data indicative of house 902/904,the system may determine that to reach delivery destination 602, thedelivery vehicle should proceed south east (i.e., down and to the leftalong the page).

In some embodiments, the resolution of the second model may differ fromthat of the first model. For example, when the delivery vehicleapproaches the delivery destination and begins collecting sensor data,the vehicle may be at a greater distance than the client computingdevice was at the time of capturing data for the first model.Accordingly, the second virtual model may be of a lower resolution thanthe first virtual model and may represent a much larger region of thedelivery destination than the first model.

In one example, when an aerial UDV first begins to approach the generalvicinity of the delivery destination 602, the UDV may be at a greaterdistance from the delivery destination than the client computing devicewas at the time of collecting data for the first virtual model.Accordingly, the sensor data received from the aerial UDV and thecorresponding second model may represent a much greater geographicregion than the first virtual model 600. For example, the second virtualmodel may represent the geographic area represented by master model 900.However, due to the distance between the aerial UDV and the deliverydestination 602, the second model may be of a lower resolution thanmodel 600.

Accordingly, the mapping operations may, based on the difference inresolution between the models and/or the difference in the size scale ofthe models, select an appropriate level of abstraction at which todetermine the mapping. For example, when the models are of approximatelythe same resolution (e.g., as are models 600 and 700), most physicalfeatures may be compared between models. However, when the second modelrepresents a larger geographic expanse than the first model and has aresolution lower than the first model, some smaller features such as,for example, windows, doors, and individual stairs might not bediscernable in the second virtual model. Accordingly, the system mayfocus on mapping between larger features such as houses, roofs ofhouses, trees, and large vegetation, among other possibilities. Forexample, as the aerial UDV approaches delivery destination 602, thesystem may determine the mapping based on the estimated spacing betweenhouses 601, 602, and 604, the shapes of the roofs of houses 601, 602,and 604, and the pattern of vegetation 620 and 622 relative to house602. As the delivery vehicle gets closer to the delivery destination,smaller features may be sufficiently resolved within the second model tomap these features to corresponding features in the first virtual model(e.g., the region around house 602 may be resolved to the level ofdetails shown in model 700).

IX. ADDITIONAL EXAMPLE IMPLEMENTATIONS

In some example implementations, the systems and operations hereindescribed may be adapted for use with existing delivery services relyingon delivery people. In particular, a second client computing deviceassociated with the delivery person may be used in place of the hardwareof the delivery vehicle. The second client computing device mayimplement functionality to guide the delivery person to the targetdrop-off spot designated by the package recipient. After arriving at thedelivery destination with the package, the delivery person may use thesecond client computing device to collect sensor data based on which thesecond model may be generated. Specifically, the second client computingdevice may display visual cues to guide the delivery person in scanningthe delivery destination to generate the second model. The second modelmay be used in combination with the first model, as previouslydescribed, to locate the target drop-off spot within the second virtualmodel.

An indication of the target drop-off spot may be displayed by the secondclient computing device in the form of a virtual-reality marker toindicate to the delivery person where to place the package. For example,the second computing device may display a live video feed from thecamera of the device (e.g., the screen of the second client computingdevice may function as a viewfinder) and, when the camera is panned overthe target drop-off spot, a virtual marker may be displayed in the videofeed indicating the target drop-off spot. Accordingly, the deliveryperson may place packages in safer and/or more convenient locations thathave been predesignated by recipients.

Although the operations are described herein in the context of objectdrop-off at a target drop-off spot within a delivery destination, theoperations may be equally adapted to object pick-up from a targetpick-up spot within a pick-up location. Namely, the user of the clientcomputing device may place an object for pickup at a target pick-up spot(e.g., a package the user wishes to return or exchange). The user mayuse the client computing device to collect sensor data based on whichthe first virtual model is generated. The user may additionally providean indication of the target pick-up spot and the object placed thereinby tapping, clicking, or otherwise selecting a representation of theobject within the first virtual model or within the captured image databased on which the first virtual model is generated. The clientcomputing device may also be programmed to allow the user to specifyparameters such as the weight of the object and the delivery destinationfor the object.

An unmanned vehicle of appropriate size and range may be dispatched topick up the object. The unmanned vehicle or a control system thereof mayperform the operations herein described to locate the target pick-upspot, pick up the object, and deliver the object to the specifieddestination for the picked-up object. Accordingly, within examples, adelivery destination is herein defined to encompass a pick-up location,a target drop-off spot is herein defined to encompass a target pick-upspot, and placement of an object at the target drop-off spot is hereindefined to encompass pick-up of the object from the target pick-up spot.

X. CONCLUSION

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its spirit and scope, as will be apparentto those skilled in the art. Functionally equivalent methods andapparatuses within the scope of the disclosure, in addition to thoseenumerated herein, will be apparent to those skilled in the art from theforegoing descriptions. Such modifications and variations are intendedto fall within the scope of the appended claims.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The exampleembodiments described herein and in the figures are not meant to belimiting. Other embodiments can be utilized, and other changes can bemade, without departing from the spirit or scope of the subject matterpresented herein. It will be readily understood that the aspects of thepresent disclosure, as generally described herein, and illustrated inthe figures, can be arranged, substituted, combined, separated, anddesigned in a wide variety of different configurations, all of which areexplicitly contemplated herein.

A block that represents a processing of information may correspond tocircuitry that can be configured to perform the specific logicalfunctions of a herein-described method or technique. Alternatively oradditionally, a block that represents a processing of information maycorrespond to a module, a segment, or a portion of program code(including related data). The program code may include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data may be stored on any type of computer readable medium suchas a storage device including a disk or hard drive or other storagemedium.

The computer readable medium may also include non-transitory computerreadable media such as computer-readable media that stores data forshort periods of time like register memory, processor cache, and randomaccess memory (RAM). The computer readable media may also includenon-transitory computer readable media that stores program code and/ordata for longer periods of time, such as secondary or persistent longterm storage, like read only memory (ROM), optical or magnetic disks,compact-disc read only memory (CD-ROM), for example. The computerreadable media may also be any other volatile or non-volatile storagesystems. A computer readable medium may be considered a computerreadable storage medium, for example, or a tangible storage device.

Moreover, a block that represents one or more information transmissionsmay correspond to information transmissions between software and/orhardware modules in the same physical device. However, other informationtransmissions may be between software modules and/or hardware modules indifferent physical devices.

The particular arrangements shown in the figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in thefigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

1. (canceled)
 2. A method comprising: receiving, from a client computingdevice, an indication of a target drop-off spot for an object within afirst virtual model of a first region, the first virtual modelindicating first physical features including three-dimensionalrelationships between objects in the first region; obtaining a secondvirtual model of a second region, the second virtual model indicatingsecond physical features including three-dimensional relationshipsbetween objects in the second region; determining, based on a comparisonbetween the first virtual model and the second virtual model, that amapping is not found between the first physical features and the secondphysical features; in response to determining that the mapping is notfound: obtaining an additional virtual model indicating third physicalfeatures including three-dimensional relationships between objects in athird region; determining a mapping between one or more of the firstphysical features and one or more of the third physical features todetermine an overlapping region between the first virtual model and theadditional virtual model; and determining, based on the overlappingregion, a position of the target drop-off spot within the additionalvirtual model; and providing instructions that cause a delivery vehicleto navigate to the target drop-off spot to place the object at thetarget drop-off spot.
 3. The method of claim 2, wherein the firstvirtual model is determined from multiple images captured by the clientcomputing device and first sensor data corresponding to the multipleimages.
 4. The method of claim 2, further comprising: receiving, fromone or more sensors on the delivery vehicle, second sensor dataindicative of the second region; and determining the second virtualmodel based on the second sensor data.
 5. The method of claim 4, whereinobtaining the additional model comprises: providing, to the deliveryvehicle, instructions to scan additional regions in response todetermining that the mapping is not found; receiving, from the one ormore sensors one the delivery vehicle, third sensor data indicative ofthe third region; and generating the additional model by updating thesecond virtual model based on the third sensor data.
 6. The method ofclaim 5, wherein the instructions to scan the additional regionscomprise instructions to scan the additional regions until a region isfound that overlaps with the first region.
 7. The method of claim 2,wherein the additional virtual model comprises a master virtual modelthat indicates physical features including three-dimensionalrelationships between objects in a general region that includes thefirst region, the second region, and the third region.
 8. The method ofclaim 7, wherein determining the mapping between one or more of thefirst physical features and one or more of the third physical featuresto determine the overlapping region between the first virtual model andthe third virtual model comprises: determining a first mapping betweenthe first virtual model and the master virtual model; determining asecond mapping between the second virtual model and the master virtualmodel; determining, based on the first mapping and the second mapping, arelative geographic positioning of the second region of the secondvirtual model with respect to the first region of the first virtualmodel; and determining, based on the relative geographic positioning ofthe second region with respect to the first region, the position of thetarget drop-off spot within the second virtual model.
 9. The method ofclaim 8, wherein providing instructions that cause the delivery vehicleto navigate to the target drop-off spot to place given object at thetarget drop-off spot comprises determining a trajectory from a currentposition of the delivery vehicle to the position of the target drop-offspot within the second virtual model.
 10. The method of claim 8, furthercomprising updating the master model using images captured by thedelivery vehicle while navigating to the target drop-off spot.
 11. Asystem comprising: a delivery vehicle; one or more sensors connected tothe delivery vehicle; and a control system configured to: receive, froma client computing device, an indication of a target drop-off spot foran object within a first virtual model of a first region, the firstvirtual model indicating first physical features includingthree-dimensional relationships between objects in the first region;obtain a second virtual model of a second region, the second virtualmodel indicating second physical features including three-dimensionalrelationships between objects in the second region; determine, based ona comparison between the first virtual model and the second virtualmodel, that a mapping is not found between the first physical featuresand the second physical features; in response to determining that themapping is not found: obtain an additional virtual model indicatingthird physical features including three-dimensional relationshipsbetween objects in a third region; determine a mapping between one ormore of the first physical features and one or more of the thirdphysical features to determine an overlapping region between the firstvirtual model and the additional virtual model; and determine, based onthe overlapping region, a position of the target drop-off spot withinthe additional virtual model; and provide instructions that cause adelivery vehicle to navigate to the target drop-off spot to place theobject at the target drop-off spot.
 12. The system of claim 11, whereinthe first virtual model is determined from multiple images captured bythe client computing device and first sensor data corresponding to themultiple images.
 13. The system of claim 11, wherein the control systemis configured to: receive, from one or more sensors on the deliveryvehicle, second sensor data indicative of the second region; anddetermine the second virtual model based on the second sensor data. 14.The system of claim 13, wherein obtaining the additional modelcomprises: providing, to the delivery vehicle, instructions to scanadditional regions in response to determining that the mapping is notfound; receiving, from the one or more sensors one the delivery vehicle,third sensor data indicative of the third region; and generating theadditional model by updating the second virtual model based on the thirdsensor data.
 15. The system of claim 14, wherein the instructions toscan the additional regions comprise instructions to scan the additionalregions until a region is found that overlaps with the first region. 16.The system of claim 11, wherein the additional virtual model comprises amaster virtual model that indicates physical features includingthree-dimensional relationships between objects in a general region thatincludes the first region, the second region, and the third region. 17.The system of claim 16, wherein determining the mapping between one ormore of the first physical features and one or more of the thirdphysical features to determine the overlapping region between the firstvirtual model and the third virtual model comprises: determining a firstmapping between the first virtual model and the master virtual model;determining a second mapping between the second virtual model and themaster virtual model; determining, based on the first mapping and thesecond mapping, a relative geographic positioning of the second regionof the second virtual model with respect to the first region of thefirst virtual model; and determining, based on the relative geographicpositioning of the second region with respect to the first region, theposition of the target drop-off spot within the second virtual model.18. The system of claim 17, wherein providing instructions that causethe delivery vehicle to navigate to the target drop-off spot to placegiven object at the target drop-off spot comprises determining atrajectory from a current position of the delivery vehicle to theposition of the target drop-off spot within the second virtual model.19. The system of claim 18, wherein the control system is configured toupdate the master model using images captured by the delivery vehiclewhile navigating to the target drop-off spot.
 20. A non-transitorycomputer readable storage medium having stored thereon instructionsthat, when executed by a computing device, cause the computing device toperform operations comprising: receiving, from a client computingdevice, an indication of a target drop-off spot for an object within afirst virtual model of a first region, the first virtual modelindicating first physical features including three-dimensionalrelationships between objects in the first region; obtaining a secondvirtual model of a second region, the second virtual model indicatingsecond physical features including three-dimensional relationshipsbetween objects in the second region; determining, based on a comparisonbetween the first virtual model and the second virtual model, that amapping is not found between the first physical features and the secondphysical features; in response to determining that the mapping is notfound: obtaining an additional virtual model indicating third physicalfeatures including three-dimensional relationships between objects in athird region; determining a mapping between one or more of the firstphysical features and one or more of the third physical features todetermine an overlapping region between the first virtual model and theadditional virtual model; and determining, based on the overlappingregion, a position of the target drop-off spot within the additionalvirtual model; and providing instructions that cause a delivery vehicleto navigate to the target drop-off spot to place the object at thetarget drop-off spot.
 21. The non-transitory computer readable storagemedium of claim 20, wherein the first virtual model is determined frommultiple images captured by the client computing device and first sensordata corresponding to the multiple images.